一些恶意日志的收集

最近调研了一些恶意日志，可以用于研究异常检测。

KDD99

KDD CUP 99 dataset 就是KDD竞赛在1999年举行时采用的数据集。1998年美国国防部高级规划署（DARPA）在MIT林肯实验室进行了一项入侵检测评估项目收集而来的数据。
内容类型: 网络流量，主机行为
是否特征化: 是
适用范围: 主机入侵检测，异常流量监控

数据来源 http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html

1998年美国国防部高级规划署（DARPA）在MIT林肯实验室进行了一项入侵检测评估项目。林肯实验室建立了模拟美国空军局域网的一个网络环境，收集了9周时间的 TCPdump(*) 网络连接和系统审计数据，仿真各种用户类型、各种不同的网络流量和攻击手段，使它就像一个真实的网络环境。这些TCPdump采集的原始数据被分为两个部分：7周时间的训练数据大概包含5,000,000多个网络连接记录，剩下的2周时间的测试数据大概包含2,000,000个网络连接记录。

一个网络连接定义为在某个时间内从开始到结束的TCP数据包序列，并且在这段时间内，数据在预定义的协议下（如TCP、UDP）从源IP地址到目的IP地址的传递。每个网络连接被标记为正常（normal）或异常（attack），异常类型被细分为4大类共39种攻击类型，其中22种攻击类型出现在训练集中，另有17种未知攻击类型出现在测试集中。

4种异常类型分别是：

DOS, denial-of-service. 拒绝服务攻击，例如ping-of-death, syn flood, smurf等；
R2L, unauthorized access from a remote machine to a local machine. 来自远程主机的未授权访问，例如guessing password；
U2R, unauthorized access to local superuser privileges by a local unpivileged user. 未授权的本地超级用户特权访问，例如buffer overflow attacks；
PROBING, surveillance and probing, 端口监视或扫描，例如port-scan, ping-sweep等。

随后来自哥伦比亚大学的Sal Stolfo 教授和来自北卡罗莱纳州立大学的 Wenke Lee 教授采用数据挖掘等技术对以上的数据集进行特征分析和数据预处理，形成了一个新的数据集。该数据集用于1999年举行的KDD CUP竞赛中，成为著名的KDD99数据集。

0,tcp,http,SF,215,45076,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,1,1,0.00,0.00,0.00,0.00,1.00,0.00,0.00,0,0,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,normal.
0,tcp,http,SF,162,4528,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,2,2,0.00,0.00,0.00,0.00,1.00,0.00,0.00,1,1,1.00,0.00,1.00,0.00,0.00,0.00,0.00,0.00,normal.
0,tcp,http,SF,236,1228,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,1,1,0.00,0.00,0.00,0.00,1.00,0.00,0.00,2,2,1.00,0.00,0.50,0.00,0.00,0.00,0.00,0.00,normal.
0,icmp,ecr_i,SF,1032,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,511,511,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,255,1.00,0.00,1.00,0.00,0.00,0.00,0.00,0.00,smurf.
0,icmp,ecr_i,SF,1032,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,511,511,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,255,1.00,0.00,1.00,0.00,0.00,0.00,0.00,0.00,smurf.
0,icmp,ecr_i,SF,1032,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,511,511,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,255,1.00,0.00,1.00,0.00,0.00,0.00,0.00,0.00,smurf.
0,icmp,ecr_i,SF,1032,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,511,511,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,255,1.00,0.00,1.00,0.00,0.00,0.00,0.00,0.00,smurf.
0,icmp,ecr_i,SF,1032,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,511,511,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,255,1.00,0.00,1.00,0.00,0.00,0.00,0.00,0.00,smurf.

特征见下：

http://blog.sina.com.cn/s/blog_707b645501010btk.html 对此有比较详细的解释

back,buffer_overflow,ftp_write,guess_passwd,imap,ipsweep,land,loadmodule,multihop,neptune,nmap,normal,perl,phf,pod,portsweep,rootkit,satan,smurf,spy,teardrop,warezclient,warezmaster.
duration: continuous.
protocol_type: symbolic.
service: symbolic.
flag: symbolic.
src_bytes: continuous.
dst_bytes: continuous.
land: symbolic.
wrong_fragment: continuous.
urgent: continuous.
hot: continuous.
num_failed_logins: continuous.
logged_in: symbolic.
num_compromised: continuous.
root_shell: continuous.
su_attempted: continuous.
num_root: continuous.
num_file_creations: continuous.
num_shells: continuous.
num_access_files: continuous.
num_outbound_cmds: continuous.
is_host_login: symbolic.
is_guest_login: symbolic.
count: continuous.
srv_count: continuous.
serror_rate: continuous.
srv_serror_rate: continuous.
rerror_rate: continuous.
srv_rerror_rate: continuous.
same_srv_rate: continuous.
diff_srv_rate: continuous.
srv_diff_host_rate: continuous.
dst_host_count: continuous.
dst_host_srv_count: continuous.
dst_host_same_srv_rate: continuous.
dst_host_diff_srv_rate: continuous.
dst_host_same_src_port_rate: continuous.
dst_host_srv_diff_host_rate: continuous.
dst_host_serror_rate: continuous.
dst_host_srv_serror_rate: continuous.
dst_host_rerror_rate: continuous.
dst_host_srv_rerror_rate: continuous.

VAST Challenge 2013: Mini-Challenge 3数据集

http://www.vacommunity.org/VAST+Challenge+2013%3A+Mini-Challenge+3

Challenge 2013是IEEE Visualization 举办的可视分析挑战赛VAST Challenge 2013 中关于网络安全数据可视分析的竞赛数据集，该数据集提供了某虚构的跨国公司内部网络两周的运行日志，日志类型有3种，分别是网络流量Netflow日志数据和Big Brother 网络健康和状态数据，日志包括：第一、二周的Netflow和Big Brother日志，第二周的入侵预防系统日志数据，通过日志的分析可以找出网络中存在的异常，网络包含的主机和服务器约1100 台，原始日志量接近10 GB，记录数超过9000万行

比赛提出的任务

了解两周内在您的网络上发生的事件，使用的所有数据来识别最多12个事件并尽可能地描述它们。

其中IP和对应的hostname也给出了（一共1100+条）：

#The following are the IPs on the Big Marketing internal network.
#Column 1:  IP
#Column 2:  Host Name. Hosts with a "WSS" prefix are user workstations. "Administrator" is the administrator workstation. Others are servers.
#Column 3 (Optional): Comments
172.10.0.2 DC01.BIGMKT1.COM	Domain controller
172.10.0.3 MAIL01.BIGMKT1.COM	SMTP
172.10.0.4 WEB01.BIGMKT1.COM	HTTP
172.10.0.5 WEB01A.BIGMKT1.COM	HTTP
172.10.0.9 WEB01B.BIGMKT1.COM	HTTP
172.10.0.7 WEB01C.BIGMKT1.COM	HTTP
172.10.0.8 WEB01D.BIGMKT1.COM	HTTP
172.10.0.40 Administrator.BIGMKT1.COM
172.10.1.1 WSS1-01.BIGMKT1.COM
172.10.1.2 WSS1-02.BIGMKT1.COM
172.10.1.3 WSS1-03.BIGMKT1.COM
172.10.1.4 WSS1-04.BIGMKT1.COM
172.10.1.5 WSS1-05.BIGMKT1.COM

数据格式

IPS-syslog-week2.csv

入侵防护系统（IPS）监视和记录网络活动。当它识别出明显的恶意活动时，IPS会尝试阻止或阻止该活动。在这种情况下，Big Marketing使用启用了威胁检测机制的思科自适应安全设备型号ASA5510。 IPS配置了默认检测规则以及特定于站点的专用规则。

Message code, inthe format < ASA-n-nnnnnn > 具体的txt需要到表里进行查找

bb-week2.csv

在网络上安装了名为Big Brother的商业网络健康监控程序。大约每五分钟，每个工作站和服务器发送一次状态更新。

bbcontent这条比较长，长这样：

Wed Apr 10 07:56:14 PDT 2013 [WEB02B.BIGMKT2.COM] up: 18 days, 1 users, 38 procs, load=0%, PhysicalMem: 4GB(14%)\n\n\n\nMemory Statistics\nTotal Physical memory:           4294500352 bytes (4.00GB)\nAvailable Physical memory:       3698483200 bytes (3.45GB)\nTotal PageFile size:             8587112448 bytes (8.00GB)\nAvailable PageFile size:         8004886528 bytes (7.46GB)\nTotal Virtual memory size:       8587112448 bytes (8.00GB)\nAvailable Virtual memory size:   8006012928 bytes (7.46GB)\n\nMost active processes\n00.07%\tvmtoolsd (0x49c [1180])\n00.03%\tsvchost#3 (0x308 [776])\n00.01%\tWmiPrvSE (0x730 [1840])\n00.01%\tw3wp (0x464 [1124])\n00.01%\tSystem (0x4 [4])\n00.01%\tsvchost#2 (0x2cc [716])\n00.01%\tsvchost#1 (0x27c [636])\n00.01%\tlsass (0x1d0 [464])\n00.01%\tbbnt (0x424 [1060])\n00.00%\twinlogon (0x188 [392])\n\n\nStatus unchanged in 59.98 minutes\nStatus message received from 172.20.0.6\n

nf-week2.csv

Big Marketing捕获防火墙的网络流量，因此可以捕获从Big Marketing到互联网，或从互联网进入Big Marketing的transactions。在网络流数据中，两台计算机之间的一系列消息被组合成一个流记录。虽然每个流记录包括源和目标IP，但源和目标的指定不保证是正确的。在流收集器未捕获流中的初始事务并且将响应视为第一事务的情况下，目标IP可被标记为源IP，反之亦然。下表描述了网络流表中的字段。

登录日志

auth.log 主要是都是登录失败的日志适合用作判断是爆破登录还是正常的输错密码
内容类型: 主机行为
是否特征化：否
使用范围: 入侵检测异常流量 WAF(网站应用级入侵防御系统)